Hybrid Time-of-Flight and Imager Module

ABSTRACT

The present disclosure relates to systems and methods that provide both an image of a scene and depth information for the scene. An example system includes at least one time-of-flight (ToF) sensor and an imaging sensor. The ToF sensor and the imaging sensor are configured to receive light from a scene. The system also includes at least one light source and a controller that carries out operations. The operations include causing the at least one light source to illuminate at least a portion of the scene with illumination light according to an illumination schedule. The operations also include causing the at least one ToF sensor to provide information indicative of a depth map of the scene based on the illumination light. The operations additionally include causing the imaging sensor to provide information indicative of an image of the scene based on the illumination light.

CROSS-REFERENCE TO RELATED APPLICATION

The present application claims the benefit of U.S. patent applicationSer. No. 16/229,193 filed Dec. 21, 2018 and U.S. Patent Application No.62/712,586, filed Jul. 31, 2018, the contents of both of which areincorporated herein by reference in their entirety.

BACKGROUND

Imaging sensors typically provide high quality, high-resolution,two-dimensional images of a scene, but do not typically provide depthinformation. Time-of-Flight (ToF) sensors typically providelow-resolution depth information about a scene, but can be subject tostray light “blooming” and/or provide inaccurate depth information whenimaging highly reflective or highly absorbing materials.

SUMMARY

The present disclosure beneficially combines aspects of imaging sensorsand ToF sensors to provide more accurate, higher-resolution depthinformation.

In a first aspect, a system is provided. The system includes at leastone time-of-flight (ToF) sensor and an imaging sensor. The at least oneToF sensor and the imaging sensor are configured to receive light from ascene. The system also includes at least one light source. The systemfurther includes a controller that carries out operations. Theoperations include causing the at least one light source to illuminateat least a portion of the scene with illumination light according to anillumination schedule. The operations include causing the at least oneToF sensor to provide information indicative of a depth map of the scenebased on the illumination light. The operations also include causing theimaging sensor to provide information indicative of an image of thescene based on the illumination light.

In a second aspect, a method is provided. The method includes causing atleast one light source to illuminate a scene with illumination lightaccording to an illumination schedule. The method also includes causinga time-of-flight (ToF) sensor to provide information indicative of adepth map of the scene based on the illumination light. The method yetfurther includes causing an imaging sensor to provide informationindicative of an image of the scene based on the illumination light.

In a third aspect, a method is provided. The method includes determiningthat a first vehicle and a second vehicle are within a thresholddistance from one another. The first vehicle and the second vehicle eachinclude respective hybrid imaging systems. The hybrid imaging systemseach include at least one time-of-flight (ToF) sensor and an imagingsensor. The at least one ToF sensor and the imaging sensor areconfigured to receive light from a scene. The hybrid imaging systemsinclude at least one light source. The method further includes adjustingat least one operating parameter of the at least one ToF sensor, theimaging sensor, or the at least one light source.

In a fourth aspect, a method is provided. The method includes providingprior information. The prior information includes three-dimensionalinformation of a scene. The method also includes causing at least onelight source to illuminate a scene with illumination light according toan illumination schedule. The method additionally includes causing atime-of-flight (ToF) sensor to provide information indicative of a depthmap of the scene based on the illumination light. The method alsoincludes causing an imaging sensor to provide information indicative ofan image of the scene based on the illumination light.

Other aspects, embodiments, and implementations will become apparent tothose of ordinary skill in the art by reading the following detaileddescription, with reference where appropriate to the accompanyingdrawings.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 illustrates a system, according to an example embodiment.

FIG. 2 illustrates an operating scenario of a system, according toexample embodiments.

FIG. 3A illustrates a vehicle, according to an example embodiment.

FIG. 3B illustrates a sensor unit, according to an example embodiment.

FIG. 3C illustrates a light source, according to an example embodiment.

FIG. 4A illustrates a sensing scenario, according to an exampleembodiment.

FIG. 4B illustrates a sensing scenario, according to an exampleembodiment.

FIG. 5 illustrates a method, according to an example embodiment.

FIG. 6 illustrates a method, according to an example embodiment.

FIG. 7 illustrates a method, according to an example embodiment.

DETAILED DESCRIPTION

Example methods, devices, and systems are described herein. It should beunderstood that the words “example” and “exemplary” are used herein tomean “serving as an example, instance, or illustration.” Any embodimentor feature described herein as being an “example” or “exemplary” is notnecessarily to be construed as preferred or advantageous over otherembodiments or features. Other embodiments can be utilized, and otherchanges can be made, without departing from the scope of the subjectmatter presented herein.

Thus, the example embodiments described herein are not meant to belimiting. Aspects of the present disclosure, as generally describedherein, and illustrated in the figures, can be arranged, substituted,combined, separated, and designed in a wide variety of differentconfigurations, all of which are contemplated herein.

Further, unless context suggests otherwise, the features illustrated ineach of the figures may be used in combination with one another. Thus,the figures should be generally viewed as component aspects of one ormore overall embodiments, with the understanding that not allillustrated features are necessary for each embodiment.

I. Overview

A hybrid imaging system could include: 1) at least one ToF sensor; 2) animaging sensor; 3) at least one light source for illuminating the sceneusing continuous, pulsed, or aperiodic illumination; and 4) acontroller, which may include a computer, a processor, and/or a DeepNeural Net. The ToF sensor and the imaging sensor may be spatiallyregistered to one another and may utilize overlapping portions of thesame optical path.

Each sensor unit of a plurality of sensor units of such a hybrid imagingsystem could be mounted on each side (or corner) of a vehicle.Respective sensor units could also be mounted in one or more spinningplatforms at various locations on the vehicle. In an example embodiment,each sensor unit may include a 180 degree field of view of a scenearound the vehicle. In some embodiments, sensor units could bepositioned on the vehicle so as to have partially overlapping fields ofview of the environment around the vehicle.

In an example embodiment, to avoid blooming or other depth informationartifacts, a plurality of ToF sensors could be associated with one ormore image sensors in a given sensor unit. The respective ToF sensorscould be spread out (e.g., spaced apart by 10 cm or more) so as toreduce the effects of blooming from specular reflections and otherbright light sources. In some embodiments, the ToF sensors could beoperated between 10-100 MHz, however other operating frequencies arecontemplated and possible. In some embodiments, the operating frequencyof the respective ToF sensor may be adjusted based on a desired maximumdepth sensing range. For instance, a ToF sensor could be operated at 20MHz for a desired depth sensing range (e.g., unambiguous range) ofapproximately 7.5 meters. In some embodiments, the ToF sensor could havea maximum desired depth sensing range of 100 meters or more.

In some embodiments, the ToF sensor could include CMOS or CCDphoto-sensitive elements (e.g., silicon PIN diodes). However, othertypes of ToF sensors and ToF sensor elements are contemplated. In somecases, the ToF sensor could be operated using various phase shift modes(e.g., a 2× or 4× phase shift).

In some embodiments, the imaging sensor could include an RGB imagingsensor, such as a megapixel-type camera sensor. The imaging sensor couldinclude a plurality of CMOS or CCD photo-sensitive elements.

In some examples, one or more light sources could be used to illuminatethe scene (or respective portions of the scene). In such scenarios, thelight sources could be modulated to provide a predetermined light pulse(or series of light pulses) that could be used in conjunction with theToF sensor to provide depth information. Additionally or alternatively,the series of light pulses (e.g., a pulse repetition rate, a pulseduration, and/or a duty cycle) could be selected so as to provide adesired exposure for the imaging sensor.

The one or more light sources could include a light strip that isdisposed along a portion of the vehicle. Additionally or alternatively,the one or more light sources could include a grid of light panels, eachsegment of which could individually provide different light pulses. Yetfurther, the one or more light sources could provide one or more lightbeams that can be moved in a point-wise and/or scanning fashion.

The one or more light sources could be operated in continuous wave (CW)and/or in pulsed (e.g., sine wave, sawtooth, or square wave) operationmode. Without limitation, the one or more light sources could include atleast one of: a laser diode, a light-emitting diode, a plasma lightsource, a strobe, a solid-state laser, a fiber laser, or another type oflight source. The one or more light sources could be configured to emitlight in the infrared wavelength range (e.g., 850, 905, and/or 940nanometers). In some embodiments, multiple illumination lightwavelengths could be used to disambiguate between multiple lightsources, etc. Additionally or alternatively, the illumination wavelengthmay be adjusted based on an amount of ambient light in the environmentand/or a time of day.

The controller could be operable to combine outputs of the respectivesensors (e.g., using sensor fusion) and/or make inferences about thethree-dimensional scene around the vehicle. For example, the controllercould make inferences to provide a grayscale or color-intensity map ofthe vehicle's surroundings. The inferences may additionally oralternatively provide information about objects in the vehicle'senvironment. In an example embodiment, the object information could beprovided at a refresh rate of 60 or 120 Hz. However, other refresh ratesare possible and contemplated.

In an example embodiment, the system could include one or more deepneural networks. The deep neural networks(s) could be utilized toprovide the inferences based on training data and/or an operatingcontext of the vehicle. In some cases, the low-resolution depthinformation and the image information may be provided to the deep neuralnetwork. Subsequently, the deep neural network could make inferencesbased on the received information and/or provide output depth maps(e.g., point clouds) at a high-resolution.

In some embodiments, two or more of: the ToF sensor, the image sensor,the light source, and the controller could be coupled to the samesubstrate. That is, the system could include a monolithic chip orsubstrate so as to provide a smaller sensor package and/or provide otherperformance improvements.

II. Example Systems

FIG. 1 illustrates a system 100, according to an example embodiment. Thesystem 100 includes at least one Time-of-Flight (ToF) sensor 110, or ToFcamera. In an example embodiment, the at least one ToF sensor 110 couldinclude a plurality of complementary metal-oxide semiconductor (CMOS) orcharge-coupled device (CCD) photosensitive elements (e.g., silicon PINdiodes). Other types of photosensitive elements could be utilized by theToF sensor 110.

In some embodiments, the at least one ToF sensor 110 could be configuredto actively estimate distances to environmental features in itsrespective field of view based on the speed of light. Namely, the ToFsensor 110 could measure the time-of-flight of a light signal (e.g., alight pulse) upon traveling between a light source (e.g., light source130) and an object in the scene. Based on estimating the time-of-flightof light pulses from a plurality of locations within a scene, a rangeimage or depth map can be built up based on the ToF sensor's field ofview. While the distance resolution can be 1 centimeter or less, thelateral resolution can be low as compared to standard 2D imagingcameras.

In some embodiments, the ToF sensor 110 can obtain images at 120 Hz orfaster. Without limitation, the ToF sensor 110 could include arange-gated imager or a direct time-of-flight imager.

The system 100 also includes at least one imaging sensor 120. In anexample embodiment, the imaging sensor 120 could include a plurality ofphotosensitive elements. In such a scenario, the plurality ofphotosensitive elements could include at least one millionphotosensitive elements. The at least one ToF sensor 110 and the atleast one imaging sensor 120 are configured to receive light from ascene.

The system 100 also includes at least one light source 130. In anexample embodiment, the at least one light source 130 could include atleast one of: a laser diode, a light-emitting diode, a plasma lightsource, a strobe light, a solid-state laser, or a fiber laser. Othertypes of light sources are possible and contemplated in the presentdisclosure. The at least one light source 130 could include a lightstrip (e.g., disposed along a portion of a vehicle). Additionally oralternatively, the at least one light source 130 could include, forexample, a grid of light panels, each segment of which couldindividually provide different light pulses. Yet further, the at leastone light source 130 could provide one or more light beams that can bemoved in a point-wise and/or scanning fashion. The at least one lightsource 130 could be operated in a continuous wave (CW) mode and/or in apulsed (e.g., sine wave, sawtooth, or square wave) operation mode.

In an example embodiment, the at least one light source 130 could beconfigured to emit infrared light (e.g., 900-1600 nanometers). However,other wavelengths of light are possible and contemplated.

The at least one light source 130 and the ToF imager 110 could betemporally synchronized. That is, a trigger signal to cause the lightsource 130 to emit light could also be provided to the ToF imager 110 asa temporal reference signal. As such, the ToF imager 110 may haveinformation about a time of the actual onset of the light emitted fromthe light source 130. Additionally or alternatively, the ToF imager 110could be calibrated based on a reference target at a known distance fromthe ToF imager 110.

In scenarios with multiple light sources and/or multiple ToF imagers,the multiple light sources could utilize time multiplexing or othertypes of signal multiplexing (e.g., frequency or code multiplexing) soas to disambiguate time-of-flight information (light pulses) obtained bya given ToF imager from the various light sources.

In some embodiments, the at least one light source 130 could beconfigured to emit light into an environment along a plurality ofemission vectors toward various target locations so as to provide adesired resolution. In such scenarios, the at least one light source 130could be operable to emit light along the plurality of emission vectorssuch that the emitted light interacts with an external environment ofthe system 100.

In an example embodiment, the respective emission vectors could includean azimuthal angle and/or an elevation angle (and/or correspondingangular ranges) with respect to a heading or location of a vehicle(e.g., vehicle 300 as illustrated and described with reference to FIG.3A). In some embodiments, light emitted by the at least one light source130 could be directed along the respective emission vectors by adjustinga movable mount and/or a movable mirror.

For example, the at least one light source 130 could emit light toward amovable mirror. By adjusting an orientation of the movable mirror, theemission vector of the light could be controllably modified. It will beunderstood that many different physical and optical techniques may beused to direct light toward a given target location. All such physicaland optical techniques for adjusting an emission vector of light arecontemplated herein.

Optionally, the system 100 may include other sensors 140. The othersensors 140 may include a LIDAR sensor, a radar sensor, or other typesof sensors. For instance, system 100 could include a Global PositioningSystem (GPS), an Inertial Measurement Unit (IMU), a temperature sensor,a speed sensor, a camera, or a microphone. In such scenarios, any of theoperational scenarios and/or methods described herein could includereceiving information from the other sensors 140 and carrying out otheroperations or method steps based, at least in part, on the informationreceived from the other sensors 140.

In an example embodiment, at least two of: the at least one ToF sensor110, the imaging sensor 120, and the at least one light source 130 couldbe coupled to a common substrate. For example, the at least one ToFsensor 110, the imaging sensor 120, and the at least one light source130 could be coupled to a vehicle. Namely, some or all elements ofsystem 100 could provide at least a portion of the object detectionand/or navigation capability of the vehicle. In example embodiments, thevehicle could be a semi-autonomous or fully-autonomous vehicle (e.g., aself-driving car). For instance, system 100 could be incorporated intovehicle 300 as illustrated and described in reference to FIGS. 3A, 4A,and 4B.

In some embodiments, system 100 could be part of a vehicle controlsystem utilized to detect and potentially identify nearby vehicles, roadboundaries, weather conditions, traffic signs and signals, andpedestrians, among other features within the environment surrounding thevehicle 300. For example, a vehicle control system may use depth mapinformation to help determine control strategy for autonomous orsemi-autonomous navigation. In some embodiments, depth map informationmay assist the vehicle control system to avoid obstacles while alsoassisting with determining proper paths for navigation.

While some examples described herein include system 100 as beingincorporated into a vehicle, it will be understood that otherapplications are possible. For example, system 100 could include, or beincorporated into, a robotic system, an aerial vehicle, a smart homedevice, a smart infrastructure system, etc.

System 100 includes a controller 150. In some embodiments, thecontroller 150 could include an on-board vehicle computer, an externalcomputer, or a mobile computing platform, such as a smartphone, tabletdevice, personal computer, wearable device, etc. Additionally oralternatively, the controller 150 can include, or could be connected to,a remotely-located computer system, such as a cloud server network. Inan example embodiment, the controller 150 may be configured to carry outsome or all of the operations, method blocks, or steps described herein.Without limitation, the controller 150 could additionally oralternatively include at least one deep neural network, another type ofmachine learning system, and/or an artificial intelligence system.

The controller 150 may include one or more processors 152 and at leastone memory 154. The processor 152 may include, for instance, amicroprocessor, an application-specific integrated circuit (ASIC), or afield-programmable gate array (FPGA). Other types of processors,circuits, computers, or electronic devices configured to carry outsoftware instructions are contemplated herein.

The memory 154 may include a non-transitory computer-readable medium,such as, but not limited to, read-only memory (ROM), programmableread-only memory (PROM), erasable programmable read-only memory (EPROM),electrically erasable programmable read-only memory (EEPROM),non-volatile random-access memory (e.g., flash memory), a solid statedrive (SSD), a hard disk drive (HDD), a Compact Disc (CD), a DigitalVideo Disk (DVD), a digital tape, read/write (R/W) CDs, R/W DVDs, etc.

The one or more processors 152 of controller 150 may be configured toexecute instructions stored in the memory 154 so as to carry out variousoperations and method steps/blocks described herein. The instructionsmay be stored in a permanent or transitory manner in the memory 154.

FIG. 2 illustrates an operating scenario 200 of the system 100,according to example embodiments. While the operating scenario 200illustrates certain operations or blocks being in a certain order andbeing carried out by certain elements of system 100, it will beunderstood that other functions, orders of operations, and/or timingarrangements are contemplated herein.

Block 210 may include the controller 150 causing the at least one lightsource 130 to illuminate at least a portion of the scene withillumination light according to an illumination schedule. Theillumination schedule could include, for example, at least one of: apredetermined light pulse repetition rate, a predetermined light pulseduration, a predetermined light pulse intensity, or a predeterminedlight pulse duty cycle. Other ways to convey desired aspects of theillumination light are contemplated herein.

In an example embodiment, instruction 212 could include, for example, asignal from the controller 150 to the light source 130 at time to. Theinstruction 212 could be indicative of the illumination schedule, anillumination level, or an illumination direction or sector, among otherexamples.

In response to receiving the instruction 212, the light source 130 couldcarry out block 214 to illuminate the scene according to theillumination schedule. As an example, the light source 130 couldilluminate one or more light-emitter elements, which could belight-emitting diodes (LEDs), lasers, strobe light, or another type oflight source. Such light-emitter elements could be illuminated accordingto the illumination schedule (e.g., illuminated for a desired time,illuminated at a desired frequency and duty cycle, etc.).

Block 220 includes causing the at least one ToF sensor 110 to provideinformation indicative of a depth map of the scene based on theillumination light. For example, at time block 220 could includeproviding an instruction 222 from the controller 150 to the ToF sensor110. The instruction 222 could include a signal to trigger a depthmapping function of the ToF sensor 110. Additionally or alternatively,the instruction 222 could include information indicative of a desiredfield of view for scanning, a desired range for scanning, a desiredresolution, and/or other desired aspects of the depth map and/or ToFsensor scan.

Block 224 could include the ToF sensor 110 obtaining a depth map based,at least in part, on the illumination of the scene from the light source130. That is, in response to receiving the instruction 222, the ToFsensor 110 may carry out a depth-mapping scan of a field of view of ascene. In an example embodiment, the ToF sensor 110 could be operatedbetween 10-100 MHz, however other operating frequencies are possible. Insome embodiments, the operating frequency of the ToF sensor 110 may beadjusted based on a desired maximum depth sensing range. For instance,the ToF sensor 110 could be operated at 20 MHz for a desired depthsensing range of approximately 7.5 meters. In some embodiments, the ToFsensor 110 could have a maximum desired depth sensing range of 100meters or more. In some embodiments that involve multiple ToF sensors,the ToF sensors could be configured to and/or instructed to carry outdepth-mapping scans of different fields of view of the scene and/or overdifferent distance ranges.

At time t₂, upon obtaining the depth map according to block 224, the ToFsensor 110 could provide information 226 to the controller 150. Theinformation 226 may be indicative of the depth map of the scene. Forexample, the information 226 could include a distance-based point map ofthe scene. Additionally or alternatively, the information 226 couldinclude a surface map of objects determined within the scene. Othertypes of information 226 are possible and contemplated.

Block 230 includes causing the imaging sensor 120 to provide informationindicative of an image of the scene based on the illumination lightprovided by the light source 130. As an example, at time t₃, thecontroller 150 could provide an instruction 232 to the imaging sensor120. The instruction 232 could include a signal for triggering an imagecapture function of the imaging sensor 120. Furthermore, the instruction232 could include information regarding a desired exposure, ambientlighting level, ambient lighting color temperature, time of day, etc.While t₁ and t₃ are illustrated in FIG. 2 as being different, in someembodiments, times t₁ and t₃ could be similar or identical. That is, insome embodiments, at least some portions of the depth mapping and imagecapture processes could be triggered and conducted in parallel.

Block 234 includes, in response to receiving the instruction 232, theimaging sensor 120 obtaining an image of the scene. In other words,instruction 232 could trigger a physical shutter mechanism or a digitalshutter so as to initiate an image capture process.

Upon capturing the image, at time t₄, the image sensor 120 could provideinformation 236 to the controller 150. The information 236 couldinclude, for example, the captured image as well as other information,such as metadata regarding the captured image (e.g., exposure time,aperture setting, imager sensitivity (ISO), field of view extents,etc.). In some embodiments, the information 236 could include RAW imagedata, however other uncompressed and compressed image data formats (BMP,JPEG, GIF, PNG, TIFF, etc.) are possible and contemplated.

Block 240 could include determining a high-resolution depth map of thescene based on the depth map of the scene (e.g., information 226) andthe image of the scene (e.g., information 236). In an exampleembodiment, the depth map information 226 and the image information 236could be compared and/or correlated using various image processingalgorithms. Such algorithms may include, without limitation, texturesynthesis, image resampling algorithms, interpolation algorithms, imagesharpening algorithms, edge-detection algorithms, and image blurringalgorithms, etc. As such, the high-resolution depth map could includedepth information about the scene with a higher spatial resolution thanthat of the depth map obtained by the ToF sensor 110. In someembodiments, the spatial resolution could relate to a target resolutionat a given distance away from the system 100. Other spatial resolutions,both along a two-dimensional surface and within three-dimensional space,are possible and contemplated herein. As an example, the depth mapobtained by the ToF sensor 110 could provide a spatial resolutionbetween adjacent sampling points of 10 centimeters at a range of 20meters. The high-resolution depth map could provide a spatial resolutionof less than 5 centimeters at a range of 20 meters.

Block 250 may include determining at least one inference about the scenebased on the depth map of the scene and the image of the scene. Forexample, the controller 150 could determine at least one inference aboutthe scene based on the high-resolution depth map determined in block240. In such a scenario, the at least one inference may includeinformation about one or more objects in an environment of a vehicle oran operating context of the vehicle. In scenarios where the controller150 includes a deep neural network, block 250 could be performed, atleast in part, by the deep neural network.

While the operating scenario 200 describes various operations or blocks210, 220, 230, 240, and 250 as being carried out by the controller 150,it will be understood that at least some of the operations of operatingscenario 200 could be executed by one or more other computing devices.

While operating scenario 200 describes various operations, it will beunderstood that more or fewer operations are contemplated. For example,the operations could further include selecting an illumination schedulefrom among a plurality of possible illumination schedules so as toprovide a desired exposure for the imaging sensor 120.

FIGS. 3A, 3B, and 3C illustrate various embodiments of the system 100and its elements. FIG. 3A illustrates a vehicle 300, according to anexample embodiment. The vehicle 300 may include one or more sensorsystems 302, 304, 306, 308, 310, 354 a-d, and 356 a-d. In some examples,the one or more sensor systems 302, 304, 306, 308, and 310 could includeLIDAR and/or radar sensor units. One or more of the sensor systems 302,304, 306, 308, and 310 may be configured to rotate about an axis (e.g.,the z-axis) perpendicular to the given plane so as to illuminate anenvironment around the vehicle 300 with light pulses and/or radarenergy. Additionally or alternatively, one or more of the sensor systems302, 304, 306, 308, and 310 could include a movable mirror so as todirect emitted light pulses and/or radar energy in the environment ofthe vehicle 300. For LIDAR-based sensors, determining various aspects ofreflected light pulses (e.g., the elapsed time of flight, polarization,etc.,) may provide information about the environment as describedherein. Similarly, radar-based sensors may determine information about agiven scene based on how radar energy interacts with the environment.

In an example embodiment, sensor systems 302, 304, 306, 308, and 310 maybe configured to provide respective point cloud information or othertypes of information (e.g., maps, object databases, etc.) that mayrelate to physical objects within the environment of the vehicle 300.While vehicle 300 and sensor systems 302 and 304 are illustrated asincluding certain features, it will be understood that other types ofsensors are contemplated within the scope of the present disclosure.

FIG. 3B illustrates a front view of sensor unit 350, according to anexample embodiment. Sensor unit 350 could include a housing 352. In someembodiments, the housing 352 could be coupled to, or integrated into,the vehicle 300. In an example embodiment, the sensor unit 350 mayinclude an imaging sensor 354, which could be similar or identical toimaging sensor 120, as illustrated and described in reference to FIG. 1. Additionally, the sensor unit 350 could include a ToF sensor 356,which could be similar or identical to ToF sensor 110, as illustratedand described in reference to FIG. 1 . While FIG. 3B illustrates imagingsensor 354 and ToF sensor 356 as being disposed within a common housing352, the imaging sensor 354 and ToF sensor 356 could be disposed indifferent locations. It will be understood that other arrangements ofsuch elements are possible and contemplated herein.

FIG. 3C illustrates a light source 370, according to an exampleembodiment.

Light source 370 could include a housing 372. In some embodiments, thehousing 372 could be coupled to, or integrated into, the vehicle 300. Inan example embodiment, the light source 370 may include a plurality oflight-emitting elements 374 a-h, which could be similar or identical tolight source 130, as illustrated and described in reference to FIG. 1 .Light-emitting elements 374 a-h could be disposed in an array or inanother spatial arrangement. In an example embodiment, thelight-emitting elements 374 a-h could be light-emitting diodes (LEDs) orlaser diodes. Other types of light sources are possible andcontemplated.

The light-emitting elements 374 a-h could be configured to emit light inthe infrared (e.g., near infrared 700-1050 nm) wavelength range.However, in some embodiments, other wavelengths of light arecontemplated. In some embodiments, the light-emitting elements 374 a-hcould be configured to emit light at different wavelengths from eachother. That is, the light-emitting elements 374 a-h could be configuredto emit light at eight different wavelengths. In such scenarios, system100 and/or vehicle 300 could be configured to disambiguate light signalsemitted by discrete light-emitting elements (or between different lightsources 370) based on its wavelength. In some embodiments, themulti-color light could be received by multi-color imaging sensorsand/or multi-color ToF sensors.

In some embodiments, light-emitting elements 374 a-h could include oneor more optical elements configured to interact with the light emittedfrom the light-emitting elements 374 a-h. Without limitation, the one ormore optical elements could be configured to redirect, shape, attenuate,amplify, or otherwise adjust the emitted light. For example, the one ormore optical elements could include a mirror, an optical fiber, adiffractive optic element, an aspherical lens, a cylindrical lens, or aspherical lens. Other types of optical elements are possible andcontemplated.

In some example embodiments, the light-emitting elements 374 a-h couldbe operable so as to emit light toward different spatial sectors (e.g.,including different azimuthal angle ranges and/or elevation angleranges) of the environment around vehicle 300. Furthermore, in someembodiments, the light-emitting elements 374 a-h could be operable toemit light at different times during a given period of time. That is,each of the light-emitting elements 374 a-h could be controlled to emitlight during respective time periods over a given time span. Forexample, the light-emitting elements 374 a-h could emit light in aserial pattern (e.g., one light-emitting element lit after another in a“chase” pattern). Additionally or alternatively, one or more of thelight-emitting elements 374 a-h could emit light in a parallel fashion(e.g., several light-emitting element emitting light simultaneously).

Returning to FIG. 3A, vehicle 300 could include a plurality of sensorunits, which could be similar or identical to sensor unit 350, asillustrated and described in reference to FIG. 3B. Furthermore, therespective sensor units could each include imaging sensors 354 a-d andToF sensors 356 a-d. As illustrated, the respective pairs of imagingsensors 354 a-d and ToF sensors 356 a-d could be coupled to, orintegrated into, a front, right side, left side, and rear portion of thevehicle 300. Other mounting types and mounting locations arecontemplated for the imaging sensors 354 a-d and ToF sensors 356 a-d.For example, in some embodiments, the imaging sensors 354 a-d and ToFsensors 356 a-d could be disposed in a rotatable mount configured torotate about the z-axis so as to obtain imaging information and ToFinformation from an environment around the vehicle 300.

While sensor systems 354 a/356 a, 354 b/356 b, 354 c/356 c, and 354d/356 d are illustrated as being collocated, it will be understood thatother sensor arrangements are possible and contemplated. Furthermore,while certain locations and numbers of sensor systems are illustrated inFIGS. 3A-3C, it will be understood that different mounting locationsand/or different numbers of the various sensor systems are contemplated.

Vehicle 300 could include a plurality of light sources 370 a-d, whichcould be similar or identical to light source 130, as illustrated anddescribed in reference to FIG. 1 . As illustrated, light source 370 a-dcould be coupled to, or integrated into, a front, right side, left side,and rear portion of the vehicle 300. Other mounting types and mountinglocations are contemplated for the plurality of light sources 370 a-d.For example, in some embodiments, the light source 370 could be disposedin a rotatable mount configured to rotate about the z-axis so as to emitlight toward a controllable azimuthal angle range.

FIG. 4A-4B illustrate various sensing scenarios 400 and 420. In eachcase, for purposes of clarity, the sensing scenarios 400 and 420 mayillustrate a subset of possible spatial sectors and sensorprofiles/ranges. It will be understood that other spatial sectors arepossible and contemplated within the scope of the present disclosure.Furthermore, it will be understood that the sensing scenarios 400 and420 may illustrate only single “snapshots” in time and that spatialsectors and sensor profiles/ranges could be dynamically adjusted so asto periodically or continuously change based on, among other factors, adynamically-changing operating context of the vehicle 300.

FIG. 4A illustrates an overhead/top view of vehicle 300 in a sensingscenario 400, according to an example embodiment. Sensing scenario 400includes illuminating a front-facing sector of an environment of thevehicle 300 with illumination light 402. Namely, light source 370 acould emit light from one or more light-emitting elements so as toilluminate the front-facing sector of the vehicle 300.

The illumination light 402 could be provided according to a pulsedillumination schedule or a continuous-wave illumination schedule. Othertypes of illumination schedules are contemplated. For example, theillumination light 402 could be provided “on-demand” from controller 150or based on the operating context of the vehicle 300. As an example, theillumination light 402 could be provided in low-light conditions (e.g.,at night) or in response to determining an object in the environment ofthe vehicle 300. As a non-limiting example, another sensor system of thevehicle 300 could identify an ambiguous or unknown object (notillustrated) ahead of the vehicle 300. The ambiguous or unknown objectcould be identified for further analysis. In such a scenario, thecontroller 150 could cause the light source 370 a to provideillumination light 402 to the front-facing sector.

While FIG. 4A illustrates a front-facing sector as being illuminated, insome embodiments, the light source 370 a may be configured to adjust apointing direction of the illumination light 402. It will also beunderstood that the other light sources 370 b-d could provide similarillumination light into various spatial sectors corresponding with theirrespective positions. For example, light source 370 d could emitillumination light into a rear-facing spatial sector.

It will be understood that while illumination light 402 and spatialsectors appear as being two-dimensional in FIG. 4A-4B, three-dimensionalspatial volumes are contemplated. For example, the illumination light402 and/or spatial sectors could be defined as between an azimuthalangle range and also between a maximum elevation angle and a minimumelevation angle.

FIG. 4B illustrates an overhead/top view of the vehicle 300 in a sensingscenario 420, according to an example embodiment. Sensing scenario 420could include imaging sensor 354 a obtaining light from a field of view404. At least a portion of the light obtained by the imaging sensor 354a could be from illumination light 402 upon interaction with theenvironment of the vehicle 300. The field of view 404 could include afront-facing spatial sector of the vehicle 300. In some embodiments, thefield of view 404 of the imaging sensor 354 a could partially or fullyoverlap with the volume illuminated by illumination light 402. Based onthe light obtained from field of view 404, the imaging sensor 354 a mayprovide an image of the scene based, at least in part, on theillumination light 402.

Sensing scenario 420 also illustrates ToF sensor 356 a obtaining lightfrom a field of view 406. At least a portion of the light obtained bythe ToF sensor 356 a could be from illumination light 402 that hasinteracted with the environment of the vehicle 300. The field of view406 could include a front-facing spatial sector of the vehicle 300. Insome embodiments, the field of view 406 of the ToF sensor 356 a couldpartially or fully overlap with the volume illuminated by illuminationlight 402. Based on the light obtained from field of view 406, the ToFsensor 356 a may provide a depth map of the scene based, at least inpart, on the illumination light 402.

III. Example Methods

FIG. 5 illustrates a method 500, according to an example embodiment. Itwill be understood that the method 500 may include fewer or more stepsor blocks than those expressly illustrated or otherwise disclosedherein. Furthermore, respective steps or blocks of method 500 may beperformed in any order and each step or block may be performed one ormore times. In some embodiments, some or all of the blocks or steps ofmethod 500 may be carried out by elements of system 100. For example,some or all of method 500 could be carried out by controller 150, ToFsensor(s) 110, and/or imaging sensor(s) 120 as illustrated and describedin relation to FIG. 1 . Furthermore, method 500 may be described, atleast in part, by the operating scenario 200, as illustrated in relationto FIG. 2 . Yet further, method 500 may be carried out, at least inpart, by vehicle 300 as illustrated and described in relation to FIG.3A. Method 500 may be carried out in scenarios similar or identical toscenario 400 as illustrated and described in relation to FIGS. 4A and4B. It will be understood that other scenarios are possible andcontemplated within the context of the present disclosure.

Block 502 includes causing at least one light source to illuminate ascene with illumination light according to an illumination schedule. Inexample embodiments, the illumination schedule could include at leastone of: a predetermined light pulse repetition rate, a predeterminedlight pulse duration, a predetermined light pulse intensity, or apredetermined light pulse duty cycle.

Block 504 includes causing a time-of-flight (ToF) sensor to provideinformation indicative of a depth map of the scene based on theillumination light. In an example embodiment, the controller 150 couldcause the ToF sensor to initiate a depth scan based on the illuminationlight. In some embodiments, a clock signal or trigger signal could beprovided to the ToF sensor to synchronize it with the one or more lightpulses emitted into the environment. Upon obtaining depth mapinformation, the ToF sensor could provide information indicative of thedepth map to the controller 150 or another element of the system 100.

Block 506 includes causing an imaging sensor to provide informationindicative of an image of the scene based on the illumination light. Insome embodiments, the controller 150 could trigger a mechanical orelectronic shutter of the imaging sensor to open and obtain an image ofthe scene. Additionally or alternatively, the controller 150 couldprovide information about the scene (e.g., ambient light level, specificsectors of concern, desired resolution, time of day, etc.). Furthermore,the controller 150 or the light source 130 could provide a clock signalor trigger signal so as to synchronize the imaging sensor and lightsource. Upon obtaining the image of the scene, the imaging sensor couldprovide information indicative of the image to the controller 150 oranother element of system 100.

Additionally or alternatively, method 500 could include selecting theillumination schedule from among a plurality of possible illuminationschedules so as to provide a desired exposure for the imaging sensor.The illumination schedule could be based on a number of variables,including external light level, other light sources, angle of sun, etc.As such, method 500 could include adjusting the illumination schedulebased on an amount of ambient light (e.g., as measured from an ambientlight sensor), a time of day, and/or weather condition.

Furthermore, method 500 could include determining a high-resolutiondepth map of the scene based on the depth map of the scene and the imageof the scene.

Yet further method 500 could include determining at least one inferenceabout the scene based on the depth map of the scene and the image of thescene. In some embodiments, the at least one inference could includeinformation about one or more objects in an environment of a vehicle oran operating context of the vehicle.

In example embodiments, determining the at least one inference could beperformed by at least one deep neural network. Additionally oralternatively, some or all blocks of method 500 could be carried out bycomputing systems implementing other types of artificialintelligence-based algorithms.

While systems and methods described herein may relate to a single hybridimaging system mounted on a vehicle, it will be understood that multiplehybrid imaging systems could be mounted on a single vehicle.Furthermore, embodiments involving multiple vehicles each having one ormore respective hybrid imaging systems are contemplated and possiblewithin the context of the present disclosure. Namely, in someembodiments, each hybrid imaging system could have a differentmodulation frequency and/or temporal offset so as to minimizeinterference with one another when close to one another (e.g., within200 meters of one another or closer).

FIG. 6 illustrates a method 600, according to an example embodiment.Method 600 could include blocks or elements that are similar oridentical to corresponding elements of methods 500 or 700, asillustrated and described in reference to FIGS. 5 and 7 .

Block 602 includes determining that a first vehicle and a second vehicleare within a threshold distance from one another. In such a scenario,the first vehicle and the second vehicle each include respective hybridimaging systems. The hybrid imaging systems could be similar oridentical to system 100, as illustrated and described in reference toFIG. 1 . That is, the hybrid imaging systems could each include at leastone time-of-flight (ToF) sensor, an imaging sensor, and at least onelight source. The at least one ToF sensor and the imaging sensor areconfigured to receive light from a scene.

Block 604 includes adjusting at least one operating parameter of the atleast one ToF sensor, the imaging sensor, or the at least one lightsource.

Additionally or alternatively, a central or regional server could assignand/or adjust the respective modulation frequencies and/or temporaloffset so as to avoid interference between proximate hybrid imagingsystems. In some embodiments, the central or regional server couldmonitor one or more operating parameters of the hybrid imaging systems(e.g., modulation frequency, temporal offset, cross-talk amplitude,etc.) and/or a location of the respective vehicles associated with thehybrid imaging systems. In response to two hybrid imaging systems and/ortheir respective vehicles approaching within a threshold distance of oneanother, the central or regional server could instruct one or both ofthe hybrid imaging systems to change their modulation frequency and/or atemporal offset so as to reduce or eliminate the possibility forcross-talk interference. Additionally or alternatively, the central orregional server could maintain a database that includes an identifierassociated with each hybrid imaging systems and at least one operatingparameter associated with each hybrid imaging system (e.g., modulationfrequency and/or temporal offset). In some embodiments, in response tothe two hybrid imaging systems and/or their respective vehiclesapproaching within a threshold distance from one another, the central orregional server could compare the database and only instruct the one ormore hybrid imaging systems to adjust their operating condition(s) ifthere may be a possibility for cross-talk interference.

While a central or regional server is described above, it will beunderstood that other, decentralized systems and methods to avoidcross-talk are contemplated. For example, if a hybrid imaging systemdetects cross-talk interference being above a threshold amplitude, thehybrid imaging system could automatically change its own modulationfrequency and/or temporal offset. Additionally or alternatively, thehybrid imaging system and/or its respective vehicle could be incommunication with nearby vehicles and/or their hybrid imaging systemsin an effort to negotiate local use of modulation frequencies and/ortemporal offsets so as to minimize or eliminate cross-talk interferencebetween nearby systems. It will be understood that other ways tomitigate interference between active sensor systems are contemplated andpossible within the context of the present disclosure.

It will be understood that systems and methods described herein couldrelate to ways in which the ToF sensors and the imaging sensors could beused to improve range-finding as compared to a ToF sensor utilized inisolation. For example, one or more images from an imaging sensor couldbe compared to an initial depth map to determine range-aliased artifactsin ToF data. That is, based on such a comparison, an updated depth mapmay be provided, which may fewer range-aliased artifacts than that ofthe initial depth map.

Additionally or alternatively, various operating parameters of the ToFsensor and/or the illumination light could be controlled based on one ormore images from an imaging sensor. For example, the image(s) mayprovide information indicative of a region of interest. For instance,the region of interest could include another vehicle, a pedestrian, anobstacle, a road marker, a road sign, etc. Based on the region ofinterest in the image(s), the operating parameters of the ToF sensorand/or the illumination light could be adjusted. For example, if theregion of interest includes a pedestrian in a crosswalk, the operatingparameters (e.g., modulation frequency, illumination intensity, refreshrate, etc.) of the ToF sensor and/or the illumination light could beoptimized or otherwise adjusted so as to provide a more accurate depthmap for the region of interest. In such a scenario, the operatingparameters may be adjusted to correspond to an estimated distance of thepedestrian in the crosswalk, or a distance range, etc.

Systems and methods described herein may involve prior information aboutthe environment. Such prior information could include a high-fidelitythree-dimensional model of the local environment of a vehicle and/orwithin a scene of the image sensor or the ToF sensor. In such scenarios,the prior information could reside, at least in part, at the vehicleand/or at a central or regional server.

In some embodiments, the prior information may be utilized incombination with the image information and/or the ToF information/depthmap to better calibrate the sensors and/or to better localize thevehicle. That is, a comparison between the prior information and atleast one image or at least one depth map could help determine intrinsicand extrinsic characteristics of the image sensor and/or ToF sensor. Insuch scenarios, the determined intrinsic and/or extrinsiccharacteristics could be used to calibrate the image sensor and/or theToF sensor. Additionally or alternatively, a comparison between theprior information and the at least one image or the at least one depthmap could include aligning or registering the prior information with theat least one image or the at least one depth map. In so doing, thealignment/registration process could help determine a more-accurateabsolute position, heading, speed, or other characteristics of thevehicle and/or other aspects of its environment. In other words, theprior information could be utilized in conjunction with the at least oneimage and/or the at least one depth map to provide more accurateinformation about the vehicle than the sensor information taken alone.In such scenarios, the prior information could represent a referenceframe within which the vehicle could be localized.

FIG. 7 illustrates a method 700, according to an example embodiment.Blocks and/or elements of method 700 could be similar or identical tocorresponding elements of methods 500 or 600, as illustrated anddescribed in reference to FIGS. 5 and 6

Block 702 includes providing prior information, which includesthree-dimensional information of a scene.

Block 704 includes causing at least one light source to illuminate thescene with illumination light according to an illumination schedule.

Block 706 includes causing a time-of-flight (ToF) sensor to provideinformation indicative of a depth map of the scene based on theillumination light.

Block 708 includes causing an imaging sensor to provide informationindicative of an image of the scene based on the illumination light.

Additionally or alternatively, the prior information could be utilizedto improve depth estimation. In such a scenario, the prior informationcould be projected into the image and/or the depth map(s). Variousmethods (e.g., ray tracing, Principle Components Ordination (PCoA),Non-metric Multidimensional Scaling (NMDS), or other methods) could beused to perform the projection of three-dimensional prior informationonto the image or depth map, each of which are contemplated herein. Byprojecting the prior information into the image or depth map, depthinformation could double-checked, calibrated, verified, and/or estimatedmore accurately.

Yet further, the prior information could be utilized to performbackground subtraction. In such a scenario, the prior information couldinclude information about objects that are outside a relevant sensordepth (e.g., far away from the vehicle). In such situations, imageinformation and/or depth map information corresponding to objects thatare outside the relevant sensor depth could be ignored, discounted,deleted, and/or processed at a lower resolution than other, morerelevant, regions of the environment.

Additionally, the prior information could be used, at least in part, todetermine where retroreflective objects may be within a givenenvironment. When a vehicle (and its hybrid imaging system(s)) entersuch an environment, it can adjust operation of the hybrid imagingsystem so as to mitigate the effects of the retroreflective objects. Forinstance, the hybrid imaging system could illuminate the environmentcorresponding to a known retroreflective object at a lower intensitylevel as compared to other regions of the environment. In such ascenario, the hybrid imaging system can avoid “blooming” or “blinding”effects that can occur due to retroreflective objects. Additionally oralternatively, the hybrid imaging system may operate at a differentmodulation frequency and/or illuminate the illumination source at adifferent rate. Other ways to mitigate the effects of retroreflectorsare possible and contemplated herein.

In some embodiments, a plurality of image frames from the image sensorcould be utilized to obtain information about the scene, which could beutilized together with other information described in the presentdisclosure. For example, “optical flow” can be obtained by a pattern ofapparent motion of an object between two consecutive image frames. Theoptical flow could include, for example, a two-dimensional vector fieldthat includes the displacement of corresponding objects in the scenebetween a first image frame and a second image frame. Based on theoptical flow, distances to the objects can be inferred and/or predicted.Such distance information from the optical flow could be utilized toconstrain the range of depths estimated when combining the imageinformation and ToF information. That is, the optical flow could providerough information about depth of objects in a given scene. The roughdepth information could be used to determine operating parameters forthe ToF sensor and/or the illumination source. Additionally oralternatively, the rough depth information could be used to bound orconstrain a set of operating parameters used by the hybrid imagingsystem more generally.

The particular arrangements shown in the Figures should not be viewed aslimiting. It should be understood that other embodiments may includemore or less of each element shown in a given Figure. Further, some ofthe illustrated elements may be combined or omitted. Yet further, anillustrative embodiment may include elements that are not illustrated inthe Figures.

A step or block that represents a processing of information cancorrespond to circuitry that can be configured to perform the specificlogical functions of a herein-described method or technique.Alternatively or additionally, a step or block that represents aprocessing of information can correspond to a module, a segment, aphysical computer (e.g., a field programmable gate array (FPGA) orapplication-specific integrated circuit (ASIC)), or a portion of programcode (including related data). The program code can include one or moreinstructions executable by a processor for implementing specific logicalfunctions or actions in the method or technique. The program code and/orrelated data can be stored on any type of computer readable medium suchas a storage device including a disk, hard drive, or other storagemedium.

The computer readable medium can also include non-transitory computerreadable media such as computer-readable media that store data for shortperiods of time like register memory, processor cache, and random accessmemory (RAM). The computer readable media can also includenon-transitory computer readable media that store program code and/ordata for longer periods of time. Thus, the computer readable media mayinclude secondary or persistent long term storage, like read only memory(ROM), optical or magnetic disks, compact-disc read only memory(CD-ROM), for example. The computer readable media can also be any othervolatile or non-volatile storage systems. A computer readable medium canbe considered a computer readable storage medium, for example, or atangible storage device.

While various aspects and embodiments have been disclosed herein, otheraspects and embodiments will be apparent to those skilled in the art.Embodiments of the present disclosure may thus relate to one of theenumerated example embodiments (EEEs) listed below.

EEE 1 is a system comprising:

at least one time-of-flight (ToF) sensor;

an imaging sensor, wherein the at least one ToF sensor and the imagingsensor are configured to receive light from a scene;

at least one light source; and

a controller that carries out operations, the operations comprising:

causing the at least one light source to illuminate at least a portionof the scene with illumination light according to an illuminationschedule;

causing the at least one ToF sensor to provide information indicative ofa depth map of the scene based on the illumination light; and

causing the imaging sensor to provide information indicative of an imageof the scene based on the illumination light.

EEE 2 is the system of EEE 1, wherein the at least one ToF sensorcomprises a plurality of complementary metal-oxide semiconductor (CMOS)or charge-coupled device (CCD) photosensitive elements.

EEE 3 is the system of EEE 1, wherein the imaging sensor comprises aplurality of photosensitive elements, wherein the plurality ofphotosensitive elements comprises at least one million photosensitiveelements.

EEE 4 is the system of EEE 1, wherein the illumination schedulecomprises at least one of: a predetermined light pulse repetition rate,a predetermined light pulse duration, a predetermined light pulseintensity, or a predetermined light pulse duty cycle.

EEE 5 is the system of EEE 1, wherein the at least one light sourcecomprises at least one of: a laser diode, a light-emitting diode, aplasma light source, a strobe light, a solid-state laser, or a fiberlaser.

EEE 6 is the system of EEE 1, wherein the operations further compriseselecting an illumination schedule from among a plurality of possibleillumination schedules so as to provide a desired exposure for theimaging sensor.

EEE 7 is the system of EEE 1, wherein the operations further comprisedetermining a high-resolution depth map of the scene based on the depthmap of the scene and the image of the scene.

EEE 8 is the system of EEE 1, wherein the at least one ToF sensor, theimaging sensor, and the at least one light source are coupled to acommon substrate.

EEE 9 is the system of EEE 1, wherein the at least one ToF sensor, theimaging sensor, and the at least one light source are coupled to avehicle.

EEE 10 is the system of EEE 1, wherein the operations further comprisedetermining at least one inference about the scene based on the depthmap of the scene and the image of the scene.

EEE 11 is the system of EEE 10, wherein the at least one inferencecomprises information about one or more objects in an environment of avehicle or an operating context of the vehicle.

EEE 12 is the system of EEE 10, wherein the controller comprises atleast one deep neural network, wherein the determining the at least oneinference is performed by the at least one deep neural network.

EEE 13 is a method comprising:

causing at least one light source to illuminate a scene withillumination light according to an illumination schedule;

causing a time-of-flight (ToF) sensor to provide information indicativeof a depth map of the scene based on the illumination light; and

causing an imaging sensor to provide information indicative of an imageof the scene based on the illumination light.

EEE 14 is the method of EEE 13, wherein the illumination schedulecomprises at least one of: a predetermined light pulse repetition rate,a predetermined light pulse duration, a predetermined light pulseintensity, or a predetermined light pulse duty cycle.

EEE 15 is the method of EEE 13, further comprising selecting theillumination schedule from among a plurality of possible illuminationschedules so as to provide a desired exposure for the imaging sensor.

EEE 16 is the method of EEE 13, further comprising determining ahigh-resolution depth map of the scene based on the depth map of thescene and the image of the scene.

EEE 17 is the method of EEE 13, further comprising determining at leastone inference about the scene based on the depth map of the scene andthe image of the scene.

EEE 18 is the method of EEE 17, wherein the at least one inferencecomprises information about one or more objects in an environment of avehicle or an operating context of the vehicle.

EEE 19 is the method of EEE 17, wherein determining the at least oneinference is performed by at least one deep neural network.

EEE 20 is the method of EEE 13, further comprising adjusting theillumination schedule based on an amount of ambient light or a time ofday.

EEE 21 is the method of EEE 13 further comprising:

comparing the image of the scene and the depth map;

based on the comparison, determining at least one range-aliased artifactin the depth map; and

providing an updated depth map based on the determined at least onerange-aliased artifact.

EEE 22 is the method of EEE 13 further comprising:

determining, based on the image of the scene, a region of interest;

adjusting at least one operating parameter of the ToF sensor based on anobject within the region of interest.

EEE 23 is the method of EEE 13 further comprising:

determining, based on a plurality of images of the scene, an opticalflow representation of the scene; and

adjusting at least one operating parameter of the ToF sensor or theillumination light based on the optical flow representation of thescene.

EEE 24 is a method comprising:

determining that a first vehicle and a second vehicle are within athreshold distance from one another, wherein the first vehicle and thesecond vehicle each comprise respective hybrid imaging systems, whereinthe hybrid imaging systems each comprise:

at least one time-of-flight (ToF) sensor;

an imaging sensor, wherein the at least one ToF sensor and the imagingsensor are configured to receive light from a scene; and

at least one light source; and

adjusting at least one operating parameter of the at least one ToFsensor, the imaging sensor, or the at least one light source.

EEE 25 is the method of EEE 24, wherein adjusting the at least oneoperating parameter comprises a server adjusting a modulation frequencyof at least one ToF sensor or adjusting a temporal offset of the atleast one ToF sensor so as to reduce cross-talk between the respectivehybrid imaging systems.

EEE 26 is the method of EEE 25, wherein the server maintains a databaseof at least one operating parameter for each hybrid imaging systemassociated with respective vehicles within a given region.

EEE 27 is a method comprising:

providing prior information, wherein the prior information comprisesthree-dimensional information of a scene;

causing at least one light source to illuminate the scene withillumination light according to an illumination schedule;

causing a time-of-flight (ToF) sensor to provide information indicativeof a depth map of the scene based on the illumination light; and

causing an imaging sensor to provide information indicative of an imageof the scene based on the illumination light.

EEE 28 is the method of EEE 27, further comprising:

comparing the prior information to at least one of the depth map or theimage of the scene; and

based on the comparison, determine a localized position of a vehicle.

EEE 29 is the method of EEE 27, further comprising:

comparing the prior information to at least one of the depth map or theimage of the scene; and

based on the comparison, determine a calibration condition of the imagesensor or the ToF sensor.

EEE 30 is the method of EEE 27, further comprising:

projecting the prior information into or onto at least one of the depthmap or the image of the scene; and

based on the projection, determine a localized position of a vehicle.

EEE 31 is the method of EEE 27, further comprising:

determining a background portion of the prior information; and

subtracting or ignoring at least a portion of the depth map or the imageof the scene corresponding to the background portion.

EEE 32 is the method of EEE 27, further comprising:

determining at least one retroreflective object based on the priorinformation; and

while scanning a portion of the scene corresponding to the at least oneretroreflective object, adjusting at least one operating parameter ofthe ToF sensor or the image sensor.

The various disclosed aspects and embodiments are for purposes ofillustration and are not intended to be limiting, with the true scopebeing indicated by the following claims.

What is claimed is:
 1. A system comprising: at least one time-of-flight(ToF) sensor; an imaging sensor, wherein the at least one ToF sensor andthe imaging sensor are configured to receive light from a scenecomprising a spatial sector, wherein the spatial sector comprises aportion of an environment around a vehicle; and a controller thatcarries out operations, the operations comprising: receivingillumination light from at least a portion of the scene, wherein theillumination light is based on an illumination schedule; causing the atleast one ToF sensor to provide information indicative of a depth map ofthe scene based on the illumination light; and causing the imagingsensor to provide information indicative of an image of the scene basedon the illumination light.
 2. The system of claim 1, wherein theillumination schedule comprises at least one of: a predetermined lightpulse repetition rate, a predetermined light pulse duration, apredetermined light pulse intensity, or a predetermined light pulse dutycycle.
 3. The system of claim 1, wherein the operations further comprisecausing at least one light source to illuminate at least the portion ofthe scene with the illumination light according to the illuminationschedule.
 4. The system of claim 1, wherein the operations furthercomprise determining a high-resolution depth map of the scene based onthe depth map of the scene and the image of the scene.
 5. The system ofclaim 1, wherein the at least one ToF sensor and the imaging sensor arecoupled to a common substrate.
 6. The system of claim 1, wherein the atleast one ToF sensor is operable to provide a distance-based point mapof the scene.
 7. The system of claim 1, wherein the operations furthercomprise determining at least one inference about the scene based on thedepth map of the scene and the image of the scene.
 8. The system ofclaim 7, wherein the at least one inference comprises information aboutone or more objects in the environment of the vehicle or an operatingcontext of the vehicle.
 9. The system of claim 7, wherein the controllercomprises at least one deep neural network, wherein the determining theat least one inference is performed by the at least one deep neuralnetwork.
 10. A method comprising: receiving illumination light from atleast a portion of a scene, wherein the illumination light is based onan illumination schedule; causing a time-of-flight (ToF) sensor toprovide information indicative of a depth map of the scene based on theillumination light; and causing an imaging sensor to provide informationindicative of an image of the scene based on the illumination light,wherein the at least on ToF sensor and the imaging sensor.
 11. Themethod of claim 10, wherein the illumination schedule comprises at leastone of: a predetermined light pulse repetition rate, a predeterminedlight pulse duration, a predetermined light pulse intensity, or apredetermined light pulse duty cycle.
 12. The method of claim 10,further comprising causing at least one light source to illuminate atleast the portion of the scene with the illumination light according tothe illumination schedule.
 13. The method of claim 10, furthercomprising determining a high-resolution depth map of the scene based onthe depth map of the scene and the image of the scene.
 14. The method ofclaim 10, further comprising determining at least one inference aboutthe scene based on the depth map of the scene and the image of thescene.
 15. The method of claim 14, wherein the at least one inferencecomprises information about one or more objects in an environment of avehicle or an operating context of the vehicle.
 16. The method of claim14, wherein determining the at least one inference is performed by atleast one deep neural network.
 17. The method of claim 10, furthercomprising adjusting the illumination schedule based on an amount ofambient light or a time of day.
 18. The method of claim 10 furthercomprising: comparing the image of the scene and the depth map; based onthe comparison, determining at least one range-aliased artifact in thedepth map; and providing an updated depth map based on the determined atleast one range-aliased artifact.
 19. The method of claim 10 furthercomprising: determining, based on the image of the scene, a region ofinterest; adjusting at least one operating parameter of the ToF sensorbased on an object within the region of interest.
 20. The method ofclaim 10 further comprising: determining, based on a plurality of imagesof the scene, an optical flow representation of the scene; and adjustingat least one operating parameter of the ToF sensor or the illuminationlight based on the optical flow representation of the scene.